Skip to content

[WIP] introduce direct source to destination type conversion#3007

Closed
jgao54 wants to merge 1 commit into
mainfrom
refactor-type-conversion-logic
Closed

[WIP] introduce direct source to destination type conversion#3007
jgao54 wants to merge 1 commit into
mainfrom
refactor-type-conversion-logic

Conversation

@jgao54

@jgao54 jgao54 commented Jun 4, 2025

Copy link
Copy Markdown
Contributor

This PR introduce a direct type conversion from PG to CH. This has the benefit of not exposing qvalue, which ideally is hidden as implementation details inside peerdb.

  • This logic is added to a shared package (flow/shared/) for easy import.
  • Added a unit test to ensure that the direct type mapping (PG type -> CH type) returns the same result as indirect type mapping (PG type -> qkind -> CH type). Currently the postgres oids are hard-coded, as I did not find a pgx lib call that allows for exhaustively iterating through the pg types, but we can turn this into an integration test and use a real postgres backend to get exhaustive pg types.
  • Currently the supported destination types is hard-coded in flow/shared, this means it's possible that ToSupportedDestinationTypesForPostgres method in flow/shared/clickhouse/type_conversion.go and supportedDestinationTypes in flow/connectors/clickhouse/type_conversion.go are out-of-sync, resulting destination type conversion getting ignored if that happens. We could add a test to ensure this is in sync as well (open to other ideas).

TODO:

  • test this e2e in CP.
  • sort through the edge cases with type modifier and custom types
  • replace old fetchAllTypeConversions with a new fetchTypeConversionsForColumns endpoint using direct type conversion
  • update peerdb ui to use new logic
  • remove qkind from column type once it's no longer needed

@jgao54 jgao54 force-pushed the refactor-type-conversion-logic branch from 945e1dc to 22258f0 Compare June 5, 2025 16:43
@jgao54 jgao54 force-pushed the refactor-type-conversion-logic branch from 22258f0 to 59dfff7 Compare June 5, 2025 16:52
@jgao54

jgao54 commented Jun 5, 2025

Copy link
Copy Markdown
Contributor Author

Closing in favor of #3011

@jgao54 jgao54 closed this Jun 5, 2025
@serprex serprex deleted the refactor-type-conversion-logic branch June 11, 2025 23:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant